Virtual staining for histology by deep learning.

In pathology and biomedical research, histology is the cornerstone method for tissue analysis. Currently, the histological workflow consumes plenty of chemicals, water, and time for staining procedures. Deep learning is now enabling digital replacement of parts of the histological staining procedure. In virtual staining, histological stains are created by training neural networks to produce stained images from an unstained tissue image, or through transferring information from one stain to another. These technical innovations provide more sustainable, rapid, and cost-effective alternatives to traditional histological pipelines, but their development is in an early phase and requires rigorous validation. In this review we cover the basic concepts of virtual staining for histology and provide future insights into the utilization of artificial intelligence (AI)-enabled virtual histology.

Platinum-based drugs induce phenotypic alterations in nucleoli and Cajal bodies in prostate cancer cells.

PURPOSE: Platinum-based drugs are cytotoxic drugs commonly used in cancer treatment. They cause DNA damage, effects of which on chromatin and cellular responses are relatively well described. Yet, the nuclear stress responses related to RNA processing are incompletely known and may be relevant for the heterogeneity with which cancer cells respond to these drugs. Here, we determine the type and extent of nuclear stress responses of prostate cancer cells to clinically relevant platinum drugs. METHODS: We study nucleolar and Cajal body (CB) responses to cisplatin, carboplatin, and oxaliplatin with immunofluorescence-based methods in prostate cancer cells. We utilize organelle-specific markers NPM, Fibrillarin, Coilin, and SMN1, and study CB-regulatory proteins FUS and TDP-43 using siRNA-mediated downregulation. RESULTS: Different types of prostate cancer cells have different sensitivities to platinum drugs. With equally cytotoxic doses, cisplatin, and oxaliplatin induce prominent nucleolar and CB stress responses while the nuclear stress phenotypes to carboplatin are milder. We find that Coilin is a stress-specific marker for platinum drug response heterogeneity. We also find that CB-associated, stress-responsive RNA binding proteins FUS and TDP-43 control Coilin and CB biology in prostate cancer cells and, further, that TDP-43 is associated with stress-responsive CBs in prostate cancer cells. CONCLUSION: Our findings provide insight into the heterologous responses of prostate cancer cells to different platinum drug treatments and indicate Coilin and TDP-43 as stress mediators in the varied outcomes. These results help understand cancer drug responses at a cellular level and have implications in tackling heterogeneity in cancer treatment outcomes.

Genomic characterization of IDH-mutant astrocytoma progression to grade 4 in the treatment setting.

As the progression of low-grade diffuse astrocytomas into grade 4 tumors significantly impacts patient prognosis, a better understanding of this process is of paramount importance for improved patient care. In this project, we analyzed matched IDH-mutant astrocytomas before and after progression to grade 4 from six patients (discovery cohort) with genome-wide sequencing, 21 additional patients with targeted sequencing, and 33 patients from Glioma Longitudinal AnalySiS cohort for validation. The Cancer Genome Atlas data from 595 diffuse gliomas provided supportive information. All patients in our discovery cohort received radiation, all but one underwent chemotherapy, and no patient received temozolomide (TMZ) before progression to grade 4 disease. One case in the discovery cohort exhibited a hypermutation signature associated with the inactivation of the MSH2 and DNMT3A genes. In other patients, the number of chromosomal rearrangements and deletions increased in grade 4 tumors. The cell cycle checkpoint gene CDKN2A, or less frequently RB1, was most commonly inactivated after receiving both chemo- and radiotherapy when compared to other treatment groups. Concomitant activating PDGFRA/MET alterations were detected in tumors that acquired a homozygous CDKN2A deletion. NRG3 gene was significantly downregulated and recurrently altered in progressed tumors. Its decreased expression was associated with poorer overall survival in both univariate and multivariate analysis. We also detected progression-related alterations in RAD51B and other DNA repair pathway genes associated with the promotion of error-prone DNA repair, potentially facilitating tumor progression. In our retrospective analysis of patient treatment and survival timelines (n = 75), the combination of postoperative radiation and chemotherapy (mainly TMZ) outperformed radiation, especially in the grade 3 tumor cohort, in which it was typically given after primary surgery. Our results provide further insight into the contribution of treatment and genetic alterations in cell cycle, growth factor signaling, and DNA repair-related genes to tumor evolution and progression.

Improved accuracy in colorectal cancer tissue decomposition through refinement of established deep learning solutions.

Hematoxylin and eosin-stained biopsy slides are regularly available for colorectal cancer patients. These slides are often not used to define objective biomarkers for patient stratification and treatment selection. Standard biomarkers often pertain to costly and slow genetic tests. However, recent work has shown that relevant biomarkers can be extracted from these images using convolutional neural networks (CNNs). The CNN-based biomarkers predicted colorectal cancer patient outcomes comparably to gold standards. Extracting CNN-biomarkers is fast, automatic, and of minimal cost. CNN-based biomarkers rely on the ability of CNNs to recognize distinct tissue types from microscope whole slide images. The quality of these biomarkers (coined 'Deep Stroma') depends on the accuracy of CNNs in decomposing all relevant tissue classes. Improving tissue decomposition accuracy is essential for improving the prognostic potential of CNN-biomarkers. In this study, we implemented a novel training strategy to refine an established CNN model, which then surpassed all previous solutions . We obtained a 95.6% average accuracy in the external test set and 99.5% in the internal test set. Our approach reduced errors in biomarker-relevant classes, such as Lymphocytes, and was the first to include interpretability methods. These methods were used to better apprehend our model's limitations and capabilities.

Deformation equivariant cross-modality image synthesis with paired non-aligned training data.

Cross-modality image synthesis is an active research topic with multiple medical clinically relevant applications. Recently, methods allowing training with paired but misaligned data have started to emerge. However, no robust and well-performing methods applicable to a wide range of real world data sets exist. In this work, we propose a generic solution to the problem of cross-modality image synthesis with paired but non-aligned data by introducing new deformation equivariance encouraging loss functions. The method consists of joint training of an image synthesis network together with separate registration networks and allows adversarial training conditioned on the input even with misaligned data. The work lowers the bar for new clinical applications by allowing effortless training of cross-modality image synthesis networks for more difficult data sets.

A Multi-Stain Breast Cancer Histological Whole-Slide-Image Data Set from Routine Diagnostics.

The analysis of FFPE tissue sections stained with haematoxylin and eosin (H&E) or immunohistochemistry (IHC) is essential for the pathologic assessment of surgically resected breast cancer specimens. IHC staining has been broadly adopted into diagnostic guidelines and routine workflows to assess the status of several established biomarkers, including ER, PGR, HER2 and KI67. Biomarker assessment can also be facilitated by computational pathology image analysis methods, which have made numerous substantial advances recently, often based on publicly available whole slide image (WSI) data sets. However, the field is still considerably limited by the sparsity of public data sets. In particular, there are no large, high quality publicly available data sets with WSIs of matching IHC and H&E-stained tissue sections from the same tumour. Here, we publish the currently largest publicly available data set of WSIs of tissue sections from surgical resection specimens from female primary breast cancer patients with matched WSIs of corresponding H&E and IHC-stained tissue, consisting of 4,212 WSIs from 1,153 patients.

Deep learning transforms colorectal cancer biomarker prediction from histopathology images.

Artificial intelligence (AI) is rapidly gaining interest in medicine, including pathological assessments for personalized medicine. In this issue of Cancer Cell, Wagner et al. demonstrate superior accuracy of transformer-based deep learning in predicting biomarker status in CRC. The work has implications for increased efficiency and accuracy in clinical diagnostics guiding treatment decisions in precision oncology.

Domain-specific transfer learning in the automated scoring of tumor-stroma ratio from histopathological images of colorectal cancer.

Tumor-stroma ratio (TSR) is a prognostic factor for many types of solid tumors. In this study, we propose a method for automated estimation of TSR from histopathological images of colorectal cancer. The method is based on convolutional neural networks which were trained to classify colorectal cancer tissue in hematoxylin-eosin stained samples into three classes: stroma, tumor and other. The models were trained using a data set that consists of 1343 whole slide images. Three different training setups were applied with a transfer learning approach using domain-specific data i.e. an external colorectal cancer histopathological data set. The three most accurate models were chosen as a classifier, TSR values were predicted and the results were compared to a visual TSR estimation made by a pathologist. The results suggest that classification accuracy does not improve when domain-specific data are used in the pre-training of the convolutional neural network models in the task at hand. Classification accuracy for stroma, tumor and other reached 96.1% on an independent test set. Among the three classes the best model gained the highest accuracy (99.3%) for class tumor. When TSR was predicted with the best model, the correlation between the predicted values and values estimated by an experienced pathologist was 0.57. Further research is needed to study associations between computationally predicted TSR values and other clinicopathological factors of colorectal cancer and the overall survival of the patients.

Deep Learning-Based Image Analysis of Liver Steatosis in Mouse Models.

The incidence of nonalcoholic fatty liver disease is a continuously growing health problem worldwide, along with obesity. Therefore, novel methods to both efficiently study the manifestation of nonalcoholic fatty liver disease and to analyze drug efficacy in preclinical models are needed. The present study developed a deep neural network-based model to quantify microvesicular and macrovesicular steatosis in the liver on hematoxylin-eosin-stained whole slide images, using the cloud-based platform, Aiforia Create. The training data included a total of 101 whole slide images from dietary interventions of wild-type mice and from two genetically modified mouse models with steatosis. The algorithm was trained for the following: to detect liver parenchyma, to exclude the blood vessels and any artefacts generated during tissue processing and image acquisition, to recognize and differentiate the areas of microvesicular and macrovesicular steatosis, and to quantify the recognized tissue area. The results of the image analysis replicated well the evaluation by expert pathologists and correlated well with the liver fat content measured by EchoMRI ex vivo, and the correlation with total liver triglycerides was notable. In conclusion, the developed deep learning-based model is a novel tool for studying liver steatosis in mouse models on paraffin sections and, thus, can facilitate reliable quantification of the amount of steatosis in large preclinical study cohorts.

The effect of neural network architecture on virtual H&E staining: Systematic assessment of histological feasibility.

Conventional histopathology has relied on chemical staining for over a century. The staining process makes tissue sections visible to the human eye through a tedious and labor-intensive procedure that alters the tissue irreversibly, preventing repeated use of the sample. Deep learning-based virtual staining can potentially alleviate these shortcomings. Here, we used standard brightfield microscopy on unstained tissue sections and studied the impact of increased network capacity on the resulting virtually stained H&E images. Using the generative adversarial neural network model pix2pix as a baseline, we observed that replacing simple convolutions with dense convolution units increased the structural similarity score, peak signal-to-noise ratio, and nuclei reproduction accuracy. We also demonstrated highly accurate reproduction of histology, especially with increased network capacity, and demonstrated applicability to several tissues. We show that network architecture optimization can improve the image translation accuracy of virtual H&E staining, highlighting the potential of virtual staining in streamlining histopathological analysis.

Unstained Tissue Imaging and Virtual Hematoxylin and Eosin Staining of Histologic Whole Slide Images.

Tissue structures, phenotypes, and pathology are routinely investigated based on histology. This includes chemically staining the transparent tissue sections to make them visible to the human eye. Although chemical staining is fast and routine, it permanently alters the tissue and often consumes hazardous reagents. On the other hand, on using adjacent tissue sections for combined measurements, the cell-wise resolution is lost owing to sections representing different parts of the tissue. Hence, techniques providing visual information of the basic tissue structure enabling additional measurements from the exact same tissue section are required. Here we tested unstained tissue imaging for the development of computational hematoxylin and eosin (HE) staining. We used unsupervised deep learning (CycleGAN) and whole slide images of prostate tissue sections to compare the performance of imaging tissue in paraffin, as deparaffinized in air, and as deparaffinized in mounting medium with section thicknesses varying between 3 and 20 µm. We showed that although thicker sections increase the information content of tissue structures in the images, thinner sections generally perform better in providing information that can be reproduced in virtual staining. According to our results, tissue imaged in paraffin and as deparaffinized provides a good overall representation of the tissue for virtually HE-stained images. Further, using a pix2pix model, we showed that the reproduction of overall tissue histology can be clearly improved with image-to-image translation using supervised learning and pixel-wise ground truth. We also showed that virtual HE staining can be used for various tissues and used with both 20× and 40× imaging magnifications. Although the performance and methods of virtual staining need further development, our study provides evidence of the feasibility of whole slide unstained microscopy as a fast, cheap, and feasible approach to producing virtual staining of tissue histology while sparing the exact same tissue section ready for subsequent utilization with follow-up methods at single-cell resolution.

Interobserver reproducibility of cribriform cancer in prostate needle biopsies and validation of International Society of Urological Pathology criteria.

AIMS: There is strong evidence that cribriform morphology indicates a worse prognosis of prostatic adenocarcinoma. Our aim was to investigate its interobserver reproducibility in prostate needle biopsies. METHODS AND RESULTS: A panel of nine prostate pathology experts from five continents independently reviewed 304 digitised biopsies for cribriform cancer according to recent International Society of Urological Pathology criteria. The biopsies were collected from a series of 702 biopsies that were reviewed by one of the panellists for enrichment of high-grade cancer and potentially cribriform structures. A 2/3 consensus diagnosis of cribriform and noncribriform cancer was reached in 90% (272/304) of the biopsies with a mean kappa value of 0.56 (95% confidence interval 0.52-0.61). The prevalence of consensus cribriform cancers was estimated to 4%, 12%, 21%, and 20% of Gleason scores 7 (3 + 4), 7 (4 + 3), 8, and 9-10, respectively. More than two cribriform structures per level or a largest cribriform mass with ≥9 lumina or a diameter of ≥0.5 mm predicted a consensus diagnosis of cribriform cancer in 88% (70/80), 84% (87/103), and 90% (56/62), respectively, and noncribriform cancer in 3% (2/80), 5% (5/103), and 2% (1/62), respectively (all P < 0.01). CONCLUSION: Cribriform prostate cancer was seen in a minority of needle biopsies with high-grade cancer. Stringent diagnostic criteria enabled the identification of cribriform patterns and the generation of a large set of consensus cases for standardisation.

STAGETOOL, a Novel Automated Approach for Mouse Testis Histological Analysis.

Spermatogenesis is a complex differentiation process that takes place in the seminiferous tubules. A specific organization of spermatogenic cells within the seminiferous epithelium enables a synchronous progress of germ cells at certain steps of differentiation on the spermatogenic pathway. This can be observed in testis cross-sections where seminiferous tubules can be classified into distinct stages of constant cellular composition (12 stages in the mouse). For a detailed analysis of spermatogenesis, these stages have to be individually observed from testis cross-sections. However, the recognition of stages requires special training and expertise. Furthermore, the manual scoring is laborious considering the high number of tubule cross-sections that have to be analyzed. To facilitate the analysis of spermatogenesis, we have developed a convolutional deep neural network-based approach named "STAGETOOL." STAGETOOL analyses histological images of 4',6-diamidine-2'-phenylindole dihydrochloride (DAPI)-stained mouse testis cross-sections at ×400 magnification, and very accurately classifies tubule cross-sections into 5 stage classes and cells into 9 categories. STAGETOOL classification accuracy for stage classes of seminiferous tubules of a whole-testis cross-section is 99.1%. For cellular level analysis the F1 score for 9 seminiferous epithelial cell types ranges from 0.80 to 0.98. Furthermore, we show that STAGETOOL can be applied for the analysis of knockout mouse models with spermatogenic defects, as well as for automated profiling of protein expression patterns. STAGETOOL is the first fluorescent labeling-based automated method for mouse testis histological analysis that enables both stage and cell-type recognition. While STAGETOOL qualitatively parallels an experienced human histologist, it outperforms humans time-wise, therefore representing a major advancement in male reproductive biology research.

Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction.

Unreliable predictions can occur when an artificial intelligence (AI) system is presented with data it has not been exposed to during training. We demonstrate the use of conformal prediction to detect unreliable predictions, using histopathological diagnosis and grading of prostate biopsies as example. We digitized 7788 prostate biopsies from 1192 men in the STHLM3 diagnostic study, used for training, and 3059 biopsies from 676 men used for testing. With conformal prediction, 1 in 794 (0.1%) predictions is incorrect for cancer diagnosis (compared to 14 errors [2%] without conformal prediction) while 175 (22%) of the predictions are flagged as unreliable when the AI-system is presented with new data from the same lab and scanner that it was trained on. Conformal prediction could with small samples (N = 49 for external scanner, N = 10 for external lab and scanner, and N = 12 for external lab, scanner and pathology assessment) detect systematic differences in external data leading to worse predictive performance. The AI-system with conformal prediction commits 3 (2%) errors for cancer detection in cases of atypical prostate tissue compared to 44 (25%) without conformal prediction, while the system flags 143 (80%) unreliable predictions. We conclude that conformal prediction can increase patient safety of AI-systems.

Detection of perineural invasion in prostate needle biopsies with deep neural networks.

The presence of perineural invasion (PNI) by carcinoma in prostate biopsies has been shown to be associated with poor prognosis. The assessment and quantification of PNI are, however, labor intensive. To aid pathologists in this task, we developed an artificial intelligence (AI) algorithm based on deep neural networks. We collected, digitized, and pixel-wise annotated the PNI findings in each of the approximately 80,000 biopsy cores from the 7406 men who underwent biopsy in a screening trial between 2012 and 2014. In total, 485 biopsy cores showed PNI. We also digitized more than 10% (n = 8318) of the PNI negative biopsy cores. Digitized biopsies from a random selection of 80% of the men were used to build the AI algorithm, while 20% were used to evaluate its performance. For detecting PNI in prostate biopsy cores, the AI had an estimated area under the receiver operating characteristics curve of 0.98 (95% CI 0.97-0.99) based on 106 PNI positive cores and 1652 PNI negative cores in the independent test set. For a pre-specified operating point, this translates to sensitivity of 0.87 and specificity of 0.97. The corresponding positive and negative predictive values were 0.67 and 0.99, respectively. The concordance of the AI with pathologists, measured by mean pairwise Cohen's kappa (0.74), was comparable to inter-pathologist concordance (0.68 to 0.75). The proposed algorithm detects PNI in prostate biopsies with acceptable performance. This could aid pathologists by reducing the number of biopsies that need to be assessed for PNI and by highlighting regions of diagnostic interest.

miR-32 promotes MYC-driven prostate cancer.

miR-32 is an androgen receptor (AR)-regulated microRNA, expression of which is increased in castration-resistant prostate cancer (PC). We have previously shown that overexpression of miR-32 in the prostate of transgenic mice potentiates proliferation in prostate epithelium. Here, we set out to determine whether increased expression of miR-32 influences growth or phenotype in prostate adenocarcinoma in vivo. We studied transgenic mice expressing MYC oncogene (hiMYC mice) to induce tumorigenesis in the mouse prostate and discovered that transgenic overexpression of miR-32 resulted in increased tumor burden as well as a more aggressive tumor phenotype in this model. Elevated expression of miR-32 increased proliferation as assessed by Ki-67 immunohistochemistry, increased nuclear density, and higher mitotic index in the tumors. By gene expression analysis of the tumorous prostate tissue, we confirmed earlier findings that miR-32 expression regulates prostate secretome by modulating expression levels of several PC-related target genes such as Spink1, Spink5, and Msmb. Further, we identified Pdk4 as a tumor-associated miR-32 target in the mouse prostate. Expression analysis of PDK4 in human PC reveals an inverse correlation with miR-32 expression and Gleason score, a decrease in castration-resistant and metastatic tumors compared to untreated primary PC, and an association of low PDK4 expression with a shorter recurrence-free survival of patients. Although decreased PDK4 expression induces the higher metabolic activity of PC cells, induced expression of PDK4 reduces both mitotic respiration and glycolysis rates as well as inhibits cell growth. In conclusion, we show that miR-32 promotes MYC-induced prostate adenocarcinoma and identifies PDK4 as a PC-relevant metabolic target of miR-32-3p.

Spatial analysis of histology in 3D: quantification and visualization of organ and tumor level tissue environment.

Histological changes in tissue are of primary importance in pathological research and diagnosis. Automated histological analysis requires ability to computationally separate pathological alterations from normal tissue. Conventional histopathological assessments are performed from individual tissue sections, leading to the loss of three-dimensional context of the tissue. Yet, the tissue context and spatial determinants are critical in several pathologies, such as in understanding growth patterns of cancer in its local environment. Here, we develop computational methods for visualization and quantitative assessment of histopathological alterations in three dimensions. First, we reconstruct the 3D representation of the whole organ from serial sectioned tissue. Then, we proceed to analyze the histological characteristics and regions of interest in 3D. As our example cases, we use whole slide images representing hematoxylin-eosin stained whole mouse prostates in a Pten+/- mouse prostate tumor model. We show that quantitative assessment of tumor sizes, shapes, and separation between spatial locations within the organ enable characterizing and grouping tumors. Further, we show that 3D visualization of tissue with computationally quantified features provides an intuitive way to observe tissue pathology. Our results underline the heterogeneity in composition and cellular organization within individual tumors. As an example, we show how prostate tumors have nuclear density gradients indicating areas of tumor growth directions and reflecting varying pressure from the surrounding tissue. The methods presented here are applicable to any tissue and different types of pathologies. This work provides a proof-of-principle for gaining a comprehensive view from histology by studying it quantitatively in 3D.

Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge.

Artificial intelligence (AI) has shown promise for diagnosing prostate cancer in biopsies. However, results have been limited to individual studies, lacking validation in multinational settings. Competitions have been shown to be accelerators for medical imaging innovations, but their impact is hindered by lack of reproducibility and independent validation. With this in mind, we organized the PANDA challenge-the largest histopathology competition to date, joined by 1,290 developers-to catalyze development of reproducible AI algorithms for Gleason grading using 10,616 digitized prostate biopsies. We validated that a diverse set of submitted algorithms reached pathologist-level performance on independent cross-continental cohorts, fully blinded to the algorithm developers. On United States and European external validation sets, the algorithms achieved agreements of 0.862 (quadratically weighted κ, 95% confidence interval (CI), 0.840-0.884) and 0.868 (95% CI, 0.835-0.900) with expert uropathologists. Successful generalization across different patient populations, laboratories and reference standards, achieved by a variety of algorithmic approaches, warrants evaluating AI-based Gleason grading in prospective clinical trials.

Virtual reality for 3D histology: multi-scale visualization of organs with interactive feature exploration.

BACKGROUND: Virtual reality (VR) enables data visualization in an immersive and engaging manner, and it can be used for creating ways to explore scientific data. Here, we use VR for visualization of 3D histology data, creating a novel interface for digital pathology to aid cancer research. METHODS: Our contribution includes 3D modeling of a whole organ and embedded objects of interest, fusing the models with associated quantitative features and full resolution serial section patches, and implementing the virtual reality application. Our VR application is multi-scale in nature, covering two object levels representing different ranges of detail, namely organ level and sub-organ level. In addition, the application includes several data layers, including the measured histology image layer and multiple representations of quantitative features computed from the histology. RESULTS: In our interactive VR application, the user can set visualization properties, select different samples and features, and interact with various objects, which is not possible in the traditional 2D-image view used in digital pathology. In this work, we used whole mouse prostates (organ level) with prostate cancer tumors (sub-organ objects of interest) as example cases, and included quantitative histological features relevant for tumor biology in the VR model. CONCLUSIONS: Our application enables a novel way for exploration of high-resolution, multidimensional data for biomedical research purposes, and can also be used in teaching and researcher training. Due to automated processing of the histology data, our application can be easily adopted to visualize other organs and pathologies from various origins.

Artificial Intelligence for Diagnosis and Gleason Grading of Prostate Cancer in Biopsies-Current Status and Next Steps.

Diagnosis and Gleason grading of prostate cancer in biopsies are critical for the clinical management of men with prostate cancer. Despite this, the high grading variability among pathologists leads to the potential for under- and overtreatment. Artificial intelligence (AI) systems have shown promise in assisting pathologists to perform Gleason grading, which could help address this problem. In this mini-review, we highlight studies reporting on the development of AI systems for cancer detection and Gleason grading, and discuss the progress needed for widespread clinical implementation, as well as anticipated future developments. PATIENT SUMMARY: This mini-review summarizes the evidence relating to the validation of artificial intelligence (AI)-assisted cancer detection and Gleason grading of prostate cancer in biopsies, and highlights the remaining steps required prior to its widespread clinical implementation. We found that, although there is strong evidence to show that AI is able to perform Gleason grading on par with experienced uropathologists, more work is needed to ensure the accuracy of results from AI systems in diverse settings across different patient populations, digitization platforms, and pathology laboratories.